A virtual desktop infrastructure (VDI) is an infrastructure that allows a user or client device to access, via a network, a virtual machine at a remote server to process and display graphics (e.g., video graphics). The graphics are processed at the VM or remote server and then displayed remotely at the user device. For example, the user device communicates with the VM at the remote server using a Transmission Control Protocol (TCP) to display video via a web browser application. When there is no bandwidth restriction, the display quality can match or be similar to that of locally processed and displayed graphics at the device. However, traffic is often transferred in bursts and network bandwidth is limited. Further, when videos for a substantial number of users are server rendered, the overall traffic bandwidth in the network increases. When a threshold bandwidth for traffic is reached, the displayed video becomes unacceptably slow to the user. Therefore, there is a need for an improved graphics/video transfer scheme in a VDI that is adaptive to bandwidth conditions between the user and the server, e.g., to avoid unacceptable slow video display speed that affects user experience.